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ABSTRACT 

Participant observation is a classic qualitative methodology 
in which the researcher is immersed in the particular setting as a 
participant, rather than an outside observer. In this pilot study of an Even 
Start program, the methodology was modified by using actual program 
participants as observers and reporters. This enabled the crossing of certain 
language and cultural barriers to gather data that would otherwise be 
difficult to obtain, increased the sense of ownership and community among the 
adult participants, and increased understanding between staff and 
participants. It also provided information for program evaluation and change. 
The Even Start Family Literacy Program is the major family literacy 
initiative administered by the U.S. Department of Education. This pilot study 
of the evaluation approach was carried out in Avery County, North Carolina in 
2003. In the 2002-2003 year, 31 families were enrolled in the program; 65% 
were English-as-a-Second-Language families. The adult participants were 
trained as participant observers for the pilot study, and most chose to 
participate, interviewing other participants. The most interesting items to 
emerge from the analysis of the transcripts are the cultural differences 
between Hispanic and Anglo participants and Hispanic participants and the 
staff, especially concerning the use of a full-time interpreted. Findings 
show that the pilot study was successful in capturing cultural identities and 
many aspects of the program from the viewpoints of staff and participants. 
(Contains 15 references.) (SLD) 
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Even Start Participant Observation Pilot: 

Using Qualitative Methodologies to Enhance Evaluations 

Abstract 

Although quantitative methodologies are used extensively in evaluation of Even 
Start programs on the national and state level, research has determined that the results of 
these evaluations are not utilized by the local projects to improve their programs. In 
recent years, qualitative methodologies have been promoted as a way of gathering and 
recording detailed information from local participants of family literacy programs. In 
order for lessons learned in the classroom to be successfully transferred to the home and 
community, family literacy programs must reflect the social contexts of the families 
involved including diversity in culture and language. Participant observation is a classic 
qualitative methodology whereby the researcher immerses herself in the particular setting 
as a participant rather than an objective outside observer. In this Even Start pilot study, 
this methodology was modified by utilizing actual program participants as the observers 
and reporters. This enabled the crossing of certain language and cultural barriers to 
gather data that would otherwise be difficult to obtain, increased the sense of ownership 
and community among the adult participants, and increased understanding between staff 
and participants. It also provided information for program evaluation and change. 

Introduction 

The Even Start Family Literacy Program was initiated on a national level in 1988 
and is the major family literacy initiative administered by the U.S. Department of 
Education. Since the program’s inception it has been evaluated frequently on a national 
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level, and each local project is required by law to conduct independent evaluations. In 
1996-97, Abt Associates collected a sample of 122 state and local evaluations to 
determine methodologies utilized and make suggestions for improvements (St. Pierre, 
Ricciuti, & Creps, 2000). They found that in 76% of the cases, the design consisted of 
pretesting at the beginning of the year and posttesting at the end of the year. Rarely were 
results compared with other groups or national Even Start scores. Although this design 
almost always showed gains among the participants for that year, studies that utilized 
control groups often showed similar gains in groups that received no intervention (Boser, 
2002). Although program gains yield figures necessary for local and state reporting, 
local projects almost never utilize the data for changing or improving the program. The 
original vision for Even Start was as a demonstration program that changed as it found 
new and better ways of delivering family literacy services to each unique local project. 

St. Pierre, Ricciuti, & Creps, (2000) concluded: 

.. .if it is useful for project directors to have a local evaluation that concludes that 
Even Start is a success, then local evaluations can be judged helpful to local 
projects. However, if the criterion for success is that a local evaluation should be 
used to help improve an Even Start project, then this review of more than 100 
evaluations was not able to show that local studies provide much useful 
information, either to local project directors, to State Coordinators, or to the 
Department, (p. 5) 

Recommendations suggested to remediate this situation included evaluation 
budgets that would be large enough to allow evaluations beyond reporting gains in test 
scores. This would allow more appropriate goal-driven measures to be chosen that would 
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result in useful interpreted data including recommendations for local programmatic 
improvements. 

This situation was recently revisited in an article by a former Even Start evaluator 
concerning the different goals of national, state, and local evaluations (Boser, 2002). 
Many local and state stakeholders feel that the standardized test scores required for 
reporting fail to adequately capture the progress made in local programs because of the 
lack of contextual elements. The move to go beyond test scores is evident in many fields 
of educational assessment. Proponents of cognitive theory argue that learning cannot be 
effectively reduced to discrete bits of information, but occurs in a larger socially 
mediated context (Johnson, McDaniel, & Willeke, 2000). These contexts, such as Even 
Start classrooms, are not static but are construction zones where socialization and 
learning occur, and contextual cultural identities are formed (Nistler & Maiers, 2000). 
Each local Even Start program is unique in the variation of participants’ characteristics. 
Some programs are mostly rural, others urban; there are variations in participant’s social 
and economic goals, their skill levels, and even their native language and culture 
(Neuman, Caperelli & Kee, 1998). Ryan, Geissler, and Knell (1996) concur that different 
types of evaluation data are essential because evaluation questions are not the same at the 
local and federal level, and that administrative attempts to mandate practices at the local 
level through the use of quantitative indicators are usually a mistake. Boser (2002) 
states: 

In my opinion, local evaluators and state leadership believed that the national 
evaluation failed to accurately represent the conditions for program 
implementation, the diversity of programs, the diversity of program participants. 
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and the diversity and subtlety of the outcomes of Even Start programs. With 
funding dependent on how the national evaluation would be viewed by 
policymakers, this perceived failure to capture and represent program impact 
could potentially undermine the future of the program. (p.30). 

As a solution to this problem, Boser (2002) recommends a participatory, 
collaborative approach to evaluation. Collaboration involves all stakeholders including 
participants of the program in the decision making process, such as inviting them to 
committee meetings, requesting input about the local project to make it more accessible 
and successful, and involving them in project goal setting. This is similar to Fetterman’s 
(2000) Empowerment Evaluation technique. In some settings, collaboration with adult 
learners led to a feeling of ownership by the participants, which resulted in contributions 
such as community outreach, organization of new projects, and teaching group sessions. 
In other settings, collaboration has also been reported to foster parity between 
participants and staff, with all stakeholders realizing the abilities, strengths, and skills 
everyone can bring to the table to forge a successful local project (Tice, 2000). 

What collaborative, participatory methods have been used to better illustrate 
progress at the local levels? Case studies often referred to as family portraits or family 
portfolios in the family literacy field have been utilized often as a qualitative 
methodology to capture details of progress made by families in family literacy programs. 
Qualitative methodology is more likely to capture the nuances of outcomes than more 
externally determined, quantitatively oriented methodologies. The benefits of this design 
include more than just greater description of program benefits. Because of the rich detail 
of these studies, stakeholders such as community members and school personnel may 
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readily see the progress made, and local support for the program, which is one of the 
Even Start goals, increased (Boser, 2002). Family portraits also document more than one 
person’s literacy development; they can document family-wide changes in literacy 
assumptions and attitudes (HofSnan, 1995). However, opponents of the family portfolio 
method point to the problem of reliable scoring and assessment among multiple raters 
(Johnson, McDaniel, & Willeke, 2000) and the amount of classroom time required to 
complete them correctly (Johnson, Willeke, & Steiner, 1998). 

In order to provide informative evaluation results at the national, state, and local 
level, a mixed method approach is recommended to integrate the perspectives of all levels 
of stakeholders, and inform decision making in both the classroom and the legislature. 
The traditional test results yielded from utilizing such instruments as the Test of Adult 
Basic Education (TABE) and Comprehensive Adult Student Assessment System 
(CASAS) will still be important at the state and national level for quantitative analyses. 
Integrating a qualitative methodology at the local level of evaluation will yield the detail 
necessary to describe improvement to local stakeholders and inform program growth and 
change. It will also reflect the social contexts of the families involved, including 
diversity in culture and language. Participant observation is a classic qualitative 
methodology usually considered ethnographic. It has been more recently used to 
describe educational programs, to focus on professional-client relationships, and as 
program evaluation. As a participant observer, the researcher immerses herself in the 
particular setting as a part of that social group rather than an objective outside observer. 
Advantages of this methodology include collection of data in the participant’s own 
setting and a description of activities and the interpretation of those activities via 




7 



5 



Pilot 



interviews (Kurz, 1983). Kurz (1983) also lists ways in whieh partieipant observation 
may be utilized for evaluation: (a) how programs actually operate at the local level, (b) 
how a program has been implemented, (c) how individuals feel about a program, and (d) 
how individuals have been changed by a program. By utilizing the actual program 
participants as the researchers, the methodology contains a culturally sensitive approach 
that can reveal and capitalize upon the cultural knowledge of the participants that may 
otherwise stay occluded (Tillman, L. C., 2002). It also overcomes some of the more 
problematic validity and reliability issues in participant observation methodology which 
utilize an outside researcher: the presence of the observer altering the behavior of those 
being studied and outside observer biases (Kurz, 1983). 

Methodology 

The pilot study was carried out in Avery County, North Carolina in the early part 
of 2003. The Avery County Even Start program is located at the Cranberry/Freedom 
Trail Elementary School in Newland, North Carolina and has been in operation since 
1998-99. Avery County is a rural, predominantly mountainous county in western North 
Carolina on the Tennessee border. According to the North Carolina Child Advocacy 
Institute, in 2001 45% of the children in Avery County participated in the school meals 
program. The percent of the population with less than a high school education is 29.4% 
(North Carolina Rural Economic Development Center, Inc.). The Even Start program 
serves local families with infants through children in third grade and above. The Avery 
County Even Start project has pioneered some innovative approaches. The preschool 
children are served together in an early childhood classroom where the older children 
help the younger ones. Infants are allowed to stay with their mothers during adult 
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instruction. Both the adult and early childhood classrooms are multicultural with 
mountain natives and Hispanic immigrants. Transportation, breakfast, and lunch is 
provided for participants. During the 2002-2003 program year, 31 families were enrolled 
in the program; 20 (65%) of which were ESL families. 

The adult participants were trained for the pilot study on February 6, 2003. 
Inherent in the participant observer process is the task of unstructured interviewing 
(Lofland & Lofland, 1995). This involves open-ended questions that arise from the 
common experience of participation in a particular setting. Topics of the training 
included interviewing skills such as utilizing open-ended questions, probes, and 
encouraging elaboration. The trainer interviewed the Even Start director as an example, 
and then the participants were given a chance to practice generating questions and 
interviewing each other. A time for questions followed the practice session. 

Participants were encouraged to think of themselves as reporters while 
interviewing, and the study was integrated into the literacy curriculum. Each week, adult 
participants generated questions that they would like to ask other participants, family 
members of participants, or staff about the Even Start Program and the impact it has on 
their lives. The teacher reviewed questions to assure proper format (i.e., open-ended and 
broad enough to elicit embellishment on the topic), and then the class chose which three 
questions were to be asked that week. The three adult participants who authored the 
chosen questions were reporters for that week. Since involvement in the study was 
entirely voluntary, if the author was unwilling to act as a reporter a willing classmate 
could take her place. The interviews were done using a tape recorder. Once a question 
was asked it could not be asked again. It was the teacher’s responsibility to keep track of 
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the questions chosen utilizing a form developed for that purpose. Also, the reporters 
were rotated so that everyone that wanted to participate had an opportunity to do so. The 
reporters interviewed at least three willing adult participants, staff members, or 
participant family members (depending on the nature of the question) a week. As a 
further incentive, the reporter was compensated for her work utilizing the token economy 
already in place at the Avery County Even Start program. If the reporter and 
interviewee’s native language was Spanish, the interviews were done in Spanish and later 
translated for content analysis. 

Reporters utilized a specific protocol to ensure uniformity, confidentiality, and 
proper tracking. First names could be used during the interview, but all participants were 
assured that no names would be used in the report. This was done for confidentiality and 
assures frankness of disclosure by the interviewee. All interviews began with the 
following information spoken into the tape recorder: date, time, reporter name, and 
interviewee name. The questions were recorded prior to each response. There were no 
time limits set for the interviews, however the nature of this methodology should lend 
itself to further questions and discussions on the chosen topic and should also be 
recorded. On May 22, 2003, there was a pilot debriefing with some of the participants 
and staff who had participated in the pilot study in order to receive feedback on what 
went well and what could be changed. 

Results 

Interviewing was done by the participants during the months of February and 
March. Most of the interviewers elaborated on their questions during the interviews, and 
responses were recorded for over 150 questions. Approximately one-third of the 
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questions were asked by English speaking students and two-thirds by Spanish speaking 
students. This closely corresponds to the ethnic percentages of the participants in the 
program. Not all questions were asked to individuals with the same native language; 
some Hispanic students interviewed English speaking staff, some English speaking 
participants interviewed Spanish speaking students with the help of the on-site translator. 
The English speaking students asked more questions of the staff than the Spanish 
speaking students and asked more questions about recruitment and how to better 
advertise the program. The Hispanic students asked more questions about the children 
and how they were taken care of and what other students thought about the teachers. 

Because the students were asked to take on the role of a reporter, most of the 
questions asked were about the program in general or about specific aspects of it. This 
provided some insight into how participants viewed the program and when interviewing 
staff, gave them an idea of how things appeared fi'om the staffs perspective. Many of the 
questions were thoughtful and the answers thorough. For content analysis, the questions 
asked were separated into eight topics: program participation and goals, specific aspects 
of the program, recruitment, suggestions for improvements, teachers, children, ESL 
issues, and staff reflections on how students had changed. These topics are similar to 
those described by Kurz (1983). 

Many of the questions concerned how staff and participants felt about the Even 
Start program and how it was helpful to their lives and families, and one participant 
interviewed one of the children about how the program had enabled his mother to help 
with his homework. Like many Even Start projects, the adult participants have many 
different goals; some are in the program to get their GED certification, some have their 
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high school diploma and are attending community college, some want better jobs, and 
some are there to learn English. Many of the participants reported that they realize it may 
take time-even years-in order to accomplish their language, educational, and occupational 
goals. The staff were also questioned about program goals, and it is apparent that the 
staff supports and encourages individualized goals among the adults. Different aspects of 
program implementation were also discussed, such as attendance, parenting classes, early 
childhood classes, GED classes, ESL classes, and computer use. When discussing 
retention, both the participants and the staff were asked how more people in the 
community could find out about the program. Participants stated that they would or had 
recommended the program to fiiends, and made suggestions for other means of 
advertisement. 

The participants had many positive things to say about the teachers of both the 
adults and the children. They characterized the teachers as being patient, loving, caring, 
and consistent with the children. They also expressed affection for the teachers and how 
they shared not only lessons, but also their lives with them. They spoke of the fun 
activities that they did in the classroom and the field trips that they took. They also 
discussed the somewhat unique arrangement of the early childhood classroom in which 
children of different ages interact together. The Hispanic participants had much to say 
about the teachers of the adults and the children, including how education differed here 
fi'om their native country. They shared that the way they were taught to do geometry and 
mathematics in their native country was different than how they were taught in the United 
States. They also reported that the teachers in their native countries were not as 
understanding and helpful as the ones in the Even Start program. 
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Program improvement is always an important topic for any program evaluation. 
Participants asked each other and staff for ideas for how the program could be improved. 
Although some participants did not think there could be any improvements, many gave 
thoughtful answers to this question. Also, many ideas for improvement were embedded 
in answers to questions on other topics and included more training on computers, more 
input from the adults on problems with understanding, and increased hygiene for the 
children. One issue that repeatedly came up with the Hispanic students in the 
improvement questions and elsewhere was the need for a full-time interpreter. The 
staffs comments clearly indicate their belief that having the adult Hispanic students 
communicate without an interpreter part of the day is beneficial for them and similar to 
the language immersion techniques utilized with the Hispanic children. The adult 
Hispanic students however felt that a full-time interpreter in the adult classroom would 
enable them to learn more quickly by having their questions addressed immediately and 
by being better able to understand and carry out instructions. 

Participants seemed particularly eager to question the staff about the methods 
used to teach the children. They asked about the different languages used, what the 
children were learning, what discipline methods were used, their preparation for 
kindergarten, and what materials were in the classrooms. One particular question 
addressed how the early childhood teacher felt when adult students questioned or 
expressed concern about the disciplinary treatment of their children. Judging from staff 
responses, this apparently arose from specific incidents and involved some 
mi srmder standing of discipline methods utilized, and the issues were reported as having 
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been resolved. Some of the participants evidently were interested in the staffs 
perception of the incidents and found the interview a safe way to discuss it. 

The final topic concerned questions to staff about the adult participants such as 
how they related to the ESL students and how they perceived the adult participants to 
have been changed by the program. These questions captured cross-cultural differences 
and the closeness that the staff had developed with the participants (Plonski, 2003): 

Interviewer: “I understand that you basically work with Hispanic women. Can 
you share with us the difference in how you work with Hispanic and how you 
work with English speakers? Could you explain some of the differences? 

Staff Respondent: One of the differences that I see is that the Hispanic women 
seem to have more confidence immediately. Because they come to school and 
I’m one of the few people they can communicate with, they get this confidence, 
and they will talk to me because I’m the only one they can talk to. And also being 
familiar with the culture, it really hasn’t been a problem helping them to feel 
comfortable with me. And it probably also has to do with my age. I’m not older 
than they are, and they don’t feel intimidated. And with the English-speaking 
girls it seems that I have to draw them out a little bit more. Try to be their fiiend, 
try to ask them questions, try to draw them out and get them to talk to me. 
Whereas most of the Hispanic women will just open up and talk, the English- 
speaking girls seem to need to be more drawn out. 

Interviewer: What changes have you seen in the women of Even Start? 

Staff Respondent: Well, I’ve noticed that when the majority of them join Even 
Start they seem to be depressed and just need a fiiend sometime. And then within 
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a couple of months they seem much happier, more outgoing, more willing to 
participate in activities that the teachers provide. I’ve seen some that stay at home 
and watch their children all day long open up and realize they are extending more 
into the world. Basically, within a couple of months, even weeks, they are out 
there and having a good time.” 

Discussion 

The most interesting items to emerge from the analysis of the transcripts are the 
cultural differences between the Hispanic and Anglo participants and the Hispanic 
participants and the staff, particularly concerning the issue of a full-time interpreter. The 
staff are seemingly confident that ‘making’ the Hispanic participants communicate 
without an interpreter would help them learn English faster. For the Hispanic students, 
their need for a full-time interpreter is a result of their desire to perform well in the class 
by completely understanding the instructions. From comments about the differences in 
educational methods used in their native countries and the United States, it seems they 
are used to a stricter atmosphere and place a greater importance upon completing an 
activity correctly the first time. They also seem to have a great respect for their teachers 
which may initially interfere with an honest display of need from the participants. This 
corresponds to the staff comment about wanting the students to be more specific about 
the problems or questions they are having. During the debriefing, the Hispanic 
participants mentioned a hesitancy to be recorded for fear of misspeaking themselves, not 
in English but in Spanish. Correct grammatical usage seems to be of greater value to 
their culture than currently in the United States. This nervousness about being taped 
crossed cultural boundaries. It was suggested by a participant that perhaps some of the 




13 



Pilot 



families of the Anglo participants did not want to answer questions on tape because of 
sensitivity about their mountain accents. This cross-cultural common ground between the 
participants was a great discovery to them, and led to the suggestion of talking about 
nervousness and fear of being recorded during the training. 

Other issues mentioned during the debriefing included the director’s comment 
that she had initially doubted that the pilot would actually uncover program needs as in 
other methods of evaluation. She found that the questions were thought provoking and 
led her take action in areas in which she had initially hesitated, such as allowing the 
participants to utilize the computers more often. The interviewing caused not only the 
interviewers to utilize higher-order thinking skills, but the one being questioned as well. 
Both the participants and staff felt that it helped them to get to know one another 
somewhat better, but that if the process were longer, perhaps over the entire year, a much 
greater understanding could be achieved. The Hispanic participants wanted to continue 
utilizing the methodology for continual self-evaluation of their English language 
acquisition; they wanted the tapes critiqued by the English participants to correct their 
pronunciation and grammar. 



Conclusion 

As participant observation, the pilot study was successful in capturing cultural 
identities and many aspects of the program from the viewpoints of staff and participants. 
As evaluation research, the results were as rich and of greater breadth than the results of a 
typical focus group, and encompassed Kurz’s (1983) list; (a) how programs actually 
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operate at the local level, (b) how a program has been implemented, (c) how individuals 
feel about a program, and (d) how individuals have been changed by a program. 

Suggested changes include utilizing the process yearlong, thereby enabling 
participants more practice at question writing, interviewing, and being interviewed. The 
tapes could then not only be used as program evaluation but as continuous self-evaluation 
by the participants themselves as their confidence increases. Issues uncovered may be 
addressed by staff in open dialogue with participants as they occur, increasing mutual 
understanding and encouraging staff-participant interaction. Increased interaction via 
interviewing between Spanish speaking and English speaking participants, participants 
and staff, and participants with each other would also likely increase feelings of 
community and collaboration. 

As this pilot study is replicated in other settings, other suggestions and 
improvements will be made regarding the methodology, increasing its use as both an 
evaluation tool and classroom activity. 
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